Search for: All records

The development of generative language models that can create long and coherent textual outputs via autoregression has lead to a proliferation of uses and a corresponding sweep of analyses as researches work to determine the limitations of this new paradigm. Unlike humans, these ‘Large Language Models’ (LLMs) are highly sensitive to small changes in their inputs, leading to unwanted inconsistency in their behavior. One problematic inconsistency when LLMs are used to answer multiple-choice questions or analyze multiple inputs is order dependency: the output of an LLM can (and often does) change significantly when sub-sequences are swapped, despite both orderings being semantically identical. In this paper we present Set-Based Prompting, a technique that guarantees the output of an LLM will not have order dependence on a specified set of sub-sequences. We show that this method provably eliminates order dependency, and that it can be applied to any transformer-based LLM to enable text generation that is unaffected by re-orderings. Delving into the implications of our method, we show that, despite our inputs being out of distribution, the impact on expected accuracy is small, where the expectation is over the order of uniformly chosen shuffling of the candidate responses, and usually significantly less in practice. Thus, Set-Based Prompting can be used as a ‘dropped-in’ method on fully trained models. Finally, we discuss how our method’s success suggests that other strong guarantees can be obtained on LLM performance via modifying the input representations. Code is available at github.com/reidmcy/set-based-prompting. 

Citizen science projects have gained momentum in recent years and involved members of the public in ongoing scientific research. Nationally, there are an estimated 8,500 volunteers monitoring U.S. water bodies and 26 states sponsoring volunteer monitoring programs (Overdevest, Orr & Stepenuck, 2004). In Oklahoma, water quality data is collected by volunteers of Blue Thumb (BT), a state-wide program emphasizing stream protection through education and involvement of the community in monitoring local water-bodies. As the first phase of a multi-phase evaluation design, the goal of this research is to map the experiential education processes and learning outcomes of the BT program. A mixed methods research design guides this programmatic review of BT and the guiding questions for the study: (1) What attributes and processes of experiential learning are found in the Blue Thumb programs?, (2) What are the measured and intended participant learning outcomes?, and (3) How do Blue Thumb educators employ experiential pedagogies to achieve these learning outcomes? The study employed an explanatory sequential mixed methods design, with qualitative data being used to add contextual understanding to the quantitative data. 

Abstract In lakes, the production and emission of methane (CH₄) have been linked to lake trophic status. However, few studies have quantified the temporal response of lake CH₄dynamics to primary productivity at the ecosystem scale or considered how the response may vary across lakes. Here, we investigate relationships between lake CH₄dynamics and ecosystem primary productivity across both space and time using data from five lakes in northern Wisconsin, USA. From 2014 to 2019, we estimated hypolimnetic CH₄storage rates for each lake using timeseries of hypolimnetic CH₄concentration through the summer season. Across all lakes and years, hypolimnetic CH₄storage ranged from <0.001 to 7.6 mmol CH₄ m⁻² d⁻¹and was positively related to the mean summer rate of gross primary productivity (GPP). However, within‐lake temporal responses to GPP diverged from the spatial relationship, and GPP was not a significant predictor of interannual variability in CH₄storage at the lake scale. Using these data, we consider how and why temporal responses may differ from spatial patterns and demonstrate how extrapolating cross‐lake relationships for prediction at the lake scale may substantially overestimate the rate of change of CH₄dynamics in response to lake primary productivity. We conclude that future predictions of lake‐mediated climate feedbacks in response to a shifting distribution of trophic status should incorporate both varying lake responses and the temporal scale of change.